The present invention concerns an optoelectronic camera comprising an optical objective system for imaging a scene recorded by the camera as an optical image substantially in an image plane of the objective system, an optoelectronic detector device substantially provided in the image plane for detecting the optical image and on basis of the detection outputting output signals, a processor device connected with the detector device for converting and processing the output signals of the detector device in order to reproduce the detected image in digital form and possibly for displaying this in real time on a display device optionally provided in the camera and connected with the processor device, and a memory device connected with the processor device for storing the digital image for displaying on the optional display device which also may be connected with the memory device, or for storing, displaying or possible additional processing on external devices adapted for these purposes and whereto the camera temporarily or permanently is connected.
The present invention also concerns an optoelectronic camera, particularly for recording colour images and even more particularly for recording colour images in an RGB system, comprising an optical objective system for imaging a scene recorded by the camera as an optical image substantially in an image plane of the objective system, an optoelectronic detector device substantially provided in the image plane for detecting the optical image and on basis of the detection outputting output signals, a processor device connected with the detector device for converting and processing the output signals of the detector device in order to reproduce the detected image in digital form and possibly for displaying this in real time on a display device optionally provided in the camera and connected with the processor device, and a memory device connected with the processor device for storing the digital image for displaying on the optional display device which also may be connected with the memory device, or for storing, displaying or possible additional processing on external devices adapted for these purposes and whereto the camera temporarily or permanently is connected. Finally, the present invention concerns a method for digital electronic formatting of a recorded full-format optical image in an optoelectronic camera according to any of the claims 1-32 or any of the claims 33-37, wherein the recorded optical image is stored as a digital image in a memory in a processor device provided in the camera and may be displayed on a display device connected to the processor device.
Very generally the invention concerns optoelectronic cameras which are suited for recording of still images as well as cinematographic images. including video images. The optoelectronic cameras according to the invention are realized such that they can be made as cheap miniature cameras with an extremely flat form factor.
After the launch of charge-coupled components (CCD), electronic photography is being applied in almost all fields in the imaging technology from the most demanding scientific applications such as in astronomical photography with recording of still images under extremely low light intensities and to applications for mass markets such as home video and area surveillance. Up to recently optoelectronic camera devices almost without exception have been based on the use of charge-coupled components (CCD), while other types, for instance charge-injected components (CID) have gained a certain use in particular applications, mostly of scientific nature. The basis for use of CCD for detection and implementation of optoelectronic cameras is extensively discussed in scientific and commercial literature and shall in the following hence be regarded as well-known to persons skilled in the art.
Even if it has been a great success, the CCD technology in optoelectronic cameras causes a number of disadvantages which has a negative effect on the possible use of CCD and miniaturized cheap battery-driven optoelectronic cameras. The silicon-based CCD chip is relatively costly to fabricate, it requires several different drive voltages and consumes relatively much current. In the course of the last years a new class of components called active pixel sensors (APS) has appeared to be strong competitors to the CCD technology, particularly in applications which do not require the absolute maximum image quality. The APS-based optoelectronic detectors can be made with low cost by means of standard CMOS technology and permits integration of a number of functions such as light detection, signal conditioning, power supply and interfacing on the same chip. In addition to the possibility of a very low cost, low power consumption and compact physical realization, the APS detectors may be realized such that processing of the image information is obtained directly on the detector chip, including for instance thresholding, contour determination etc. For certain types of applications APS detectors may give fast random access to the selected pixels or groups of pixels, in contrast with CCD-based detectors which require serial readout of whole rows of pixels at one time.
Commercial applications of APS-based miniature device have emerged within a number of areas, either supplanting other technologies or generating wholly new products. An instance of the first is the use in surveillance cameras, an instance of the latter is the use in toys. Due to the particular properties of the APS detectors recent development has led to optoelectronic cameras with very small dimensions. Such so-called xe2x80x9con chipxe2x80x9d-cameras may be obtained commercially from a number of companies, for instance VLSI Vision, Scotland, firms in Sweden and Photobit, U.S.A. A camera which may be accomodated in fountain pen format was recently demonstrated by CSEM, Switzerland.
A common denominator for all optoelectronic camera types is an optical system which creates an acceptable optical image on the light-sensitive detector surface. This poses a problem when it is desired to miniaturize optoelectronic cameras regardless of the type of optoelectronic sensor (CCD, CID, APS, diode array . . . ) to be used. The problem becomes particularly accentuated if the available axial length (the distance along the optical axis from the front of the camera lens and therethrough to the back of the detector chip) of the camera is restricted, i.e. when it is desirable to manufacture a flat camera, as the contribution from the imaging system to this distance is the sum of the lens thickness and the back focal length (BFL), something which indicates that a lenslet or microlens with a very short axial dimension and very short focal length might be used to provide a flat camera solution. However, up to now really flat miniaturized optoelectronic cameras based on this principle have not emerged.
As is to be discussed in the following, the main reason for this substantially is not to be found in the optics used and which generates the optical image. Even if the resolution in the last instance is limited by diffraction, there is another delimiting factor which to a much larger extent finds expression in the present context, namely the restricted spatial resolution which may be obtained in the image plane, particularly with optoelectronic detector arrays. In order to better illuminate the logical step in the development which has led to the present invention, there shall in the following be given a simple basic analysis of the drawbacks of the prior art.
The quality of the optical image will depend on the lens construction and is, as mentioned, in the last instance, limited by diffraction. In order to simplify the analysis it shall be supposed that light is monochromatic green, for instance with a wavelength of 555 nm, and that the lens is very thin and diffraction-limited. The spatial resolution in the image plane is then given by
w=0.61xcex/NAxe2x80x83xe2x80x83(1)
wherein xcex is the light""s wavelength and the numerical aperture NA defined as
NA=n sinxcex1.xe2x80x83xe2x80x83(2)
Here n is the refraction index in the image space and xcex1 the half angle of the edge rays in the image space.
The resolution is in principle independent of the physical size of the lens. With realistic values for the numerical aperture NA, the resolution, however, is typically comparable to the light""s wavelength. This implies that an image which shall contain M resolved elements (M=nxny, where nx and ny is the number of resolved elements along respectively the x and y axis) must cover an area in the image plane which cannot be less than
A=M w2=nx,ny, w2.xe2x80x83xe2x80x83(3)
Here w is the resolution as defined in equation (1) above.
The field of view of the lenses in combination with the linear dimensions nx, w and ny, w for the image defines the focal length of the lens and hence the physical size of the latter. The field of view is defined by the half angle xcex8 of arrays striking the extreme edge of the optical image, at which location the linear distance to the optical axis is given by
d/2=(nx2+ny2)1/2, w/2xe2x80x83xe2x80x83(4)
Denoting the image distance by sxe2x80x2, one has
sxe2x80x2=d/(2tgxcex8)=(nx2+ny2)1/2w/(2tgxcex8)xe2x80x83xe2x80x83(5)
For typical imaging cases the focal length for the lens is only slightly less than the image distance sxe2x80x2, i.e.
xe2x80x83f≅sxe2x80x2xe2x80x83xe2x80x83(6)
If numerical values are inserted for instance w=0.5 xcexcm, nx=ny=103, i.e. M=106, xcex8=19.3xc2x0, one obtains f≅sxe2x80x2=1.01 mm.
A microlens with this focal length has typically comparable linear dimensions and it will be realized that a truly miniaturized flat camera may be within reach, and offering a resolution of 1 million pixels.
Unfortunately the requirement that the resolution w shall be 0.5 xcexcm such as defined above for the recording medium in the image plane, is very difficult to realize and far beyond what may be implemented with pixelated optoelectronic image sensors, CCD and APS detectors according to prior art has a pixel pitch at least ten times the resolution w assumed above, something which implies that the focal length and the linear dimensions of the lens shall lie in the range from 10 mm and upwards. Evidently the linear size of the camera depends explicitly of the desired image quality, i.e. on the detail of resolution in the image and of whether it is desirable with a monochromatic image or a full colour image. Hence optoelectronic cameras with dimensions along the optical axis in the range of 1 cm may be implemented. This cannot, however, be regarded as being particularly flat. Smaller dimensions are possible, but entails an impaired image quality. For instances of xe2x80x9con-chip-cameraxe2x80x9d concepts which exploit CMOS processors to manufacture optoelectronic camera devices with low costs and/or for particular purposes, reference shall be made to literature from e.g. Photobit. U.S.A; IVP, Sweden; VLSI Vision, Great Britain; CSEM, Switzerland; and IMEC, Belgium. For a review of imaging techniques with the use of CMOS-technology, reference may for instance be made to J. Zarnowski and M. Pace, xe2x80x9cImaging options expand with CMOS technologyxe2x80x9d, Laser Focus World, pp. 125-130 (June 1997).
The main purpose of the present invention is to provide an optoelectronic camera which may be used for recording still images, cinematographic images or video images with high image quality and beyond all with high spatial resolution, while the total profile of the camera appears as very flat and the drawbacks which are linked with the above-mentioned prior art are avoided, and then particularly that the image resolution scales with the physical size, particularly the axial length of the optical imaging system.
It is also the object of the invention to provide an optoelectronic camera which may be realized as a relatively thin layer, typically in the size range of 1-3 mm thickness on flat or curved surfaces.
Further it is an object of the invention to provide an optoelectronic camera with a number of specific spatial and spectral imaging characteristics, including controlled resolution of the optical image in one or more areas in the image or along one or more axes in the image plane, an extremely large field of view, including up to a global field (4xcfx80 steradians), spatially resolved chromatic or spectral analysis, full-colour images or imaging in one or more wavelength bands from ultraviolet to infrared and parallaxis-based imaging with the possibility of spatial detection and analysis.
Yet further it is also an object of the invention to provide an optoelectronic camera with imaging solutions which exploit light-detecting elements and circuits realized in optoelectronic technology on large surfaces. Such technology will allow an optoelectronic camera according to the invention to be manufactured with particular low cost. Finally it is a special object of the invention that the optoelectronic camera shall be realized with the use of thin devices based on amorphous or polycrystalline inorganic semiconductors or organic semiconductors based on polymers or oligomers. An example of the application of such material shall be components in the form of flexible plastic sheets, realized as thin cards which may be attached to flat and curved surfaces.
It is also a special object of the present invention to be able to realize ultraminiaturized optoelectronic cameras by using arrays of quasi-monochromatic microlenses as the optical active structures in the camera.
The above objects are realized according to the invention with an optoelectronic camera which is characterized in that the camera objective system is formed by an array of two or more optical active structures (L), that each optical active structure is adapted for generating optical images of the recorded scene on areas of the objective system image plane uniquely assigned to the respective optical active structure, that at least one optoelectronic detector is provided for each optical active structure in its respective assigned area or image plane, all detectors being included in the detector device of the camera, that each detector comprises at least one sensor element uniquely defining a pixel of the optical image, the area of the pixel substantially being determined by the area of the separate defining sensor element, and that each detector is adapted for defining a sample of the optical image with a number of pixels in each sample determined by a number of sensor elements in the defining detector, the digital image optimally being generated by all samples and with a spatial resolution determined by the number of pixels in distinct positions in the optical image defined by the sensor elements.
Advantageously the optical active structures in this connection are refractive structures or diffractive structures or reflective structures or combinations of such structures.
Particular it is advantageous that the refractive or diffractive structures are realized as lenslets with a diameter of at most 3 mm.
It is also advantageous when the total number of distinctly defined pixels in the optical image is equal to the total number of sensor elements in the detector device, such that a one-to-one relation between a given pixel and its defining sensor element in this case is present, whereby the digital image may be generated by a full sampling of the optical image or that the total number of distinctly defined pixels in the optical image is smaller than the total number of sensor elements in the detector device, such that a one-to-many relation between a given pixel and its defining sensor element or sensor elements in this case is present, whereby the digital image may be generated by an oversampling of the optical image. It is advantageous that the optoelectronic camera comprises one or more spatial filters provided in front of the objective system and/or between the objective system and the detector device, said spatial filter perferably being a spatial light-modulator and particular in that connection a controllable electrooptical light-modulator.
It is also advantageous when the optoelectronic camera according to the invention comprises one or more optical filter means provided in front of the objective system and/or between the objective system and the detector device. Preferably the optical filter means then may comprise separate, spectral-selective filters which separately are assigned to either each optical active structure or groups of optical active structures, or to the detector or detectors of the detector device assigned to each optical active structure.
Particularly the optical filter means may be adapted for transmitting in two or more separate wavelength bands by each spectral-selective filter transmitting in a separate wavelength band, the number of filters which transmits in each of the separate wavelength bands substantially being identical. The separate wavelength bands may then preferably be selected such that the optical filter means forms a primary colour filter means or an RGB filter means or such that the optical filter means forms a complementary colour filter means.
In some embodiments the spectral-selective filter advantageously may be realized as a strip filter which is adapted for transmitting in two or more separate wavelength bands by each strip transmitting in a given wavelength band. Preferably each strip in a strip filter may then be assigned to a respective row or column of sensor elements in the detector or detectors and each strip filter may further be realized as a primary colour filter or an RGB filter.
It may according to the invention also be advantageous that the spectral-selective filter is a mosaic filter which is adapted for transmitting in two or more separate wavelength bands by each filter segment in the mosaic filter transmitting in a given wavelength band, the number of filter segments which transmits in each of the wavelength bands substantially being identical, and preferably each filter segments in a mosaic filter then assigned to a respective sensor element or respective sensor elements in the detector or detectors. Particularly may then each mosaic filter preferably be a complementary colour filter.
It is according to the invention advantageous that the detector device comprises detectors realized in one of the following technologies, viz. CCD (charge-coupled device) technology, CID (charge-injected device) technology, APS (active pixel sensor) technology or PMSA (sensor array in passive matrix) technology. Wherein the detector is realized in PMSA technology it is according to the invention advantageous that the detector is realized as a thin-film component or a hybrid component, and that the detector is adapted for parallel read-out of the output signals from the sensor elements over a passive electrode array for unique addressing of each separate sensor element, as the detector in this case preferably may be made wholly or partly of organic semiconducting or electrical isolating materials, including plastic materials and semiconducting oligomers or polymers.
It is according to the invention advantageous that the optical active structures are realized with a determined chromatic aberration or dispersion, such that each optical active structure for two or more separate wavelength bands spectral-selectively generates the optical image in each wavelength band substantially on corresponding substantially congruent image planes superpositioned spaced apart in the ray direction, and that for each optical active structure in each of these image planes a detector for spectral selective detection of the optical image is provided such that for each optical active structure on each image plane a sample in the spatial domain and a sample in the frequency domain are defined, the resolution in the frequency domain substantially being determined by the number of separate wavelength bands with a respective assigned detector, whereby the optical image detected by the detector device may be generated as a multispectral digital colour image with the use of a selected suitable colour system. In that connection it may for each optical active structure preferably be provided three separate superpositioned detectors, respectively in the image plane for three separate wavelength bands assigned to a three-colour system.
Further the above-mentioned objects are realized according to the present invention with an optoelectronic camera which is characterized in that that the camera objective system is formed by an array of two or more optical active structures, that each optical active structure has a determined chromatic aberration or dispersion such that the location of its focal point depends on the wavelength of the light, that each optical structure is adapted for generating spectral-selectively optical images of the recorded scene on areas of three separate superpositioned image planes of the objective system, said areas being uniquely assigned to respective optical active structures, a first image plane forming a first optical image in a wavelength band in the blue portion of the spectrum, and a second image plane a second optical image in a wavelength band in the green portion of the spectrum and a third image plane a third optical image in a wavelength band in the red portion of the spectrum, that for each optical active structure an optoelectronic detector is provided in each of the respective assigned image planes for detection of the optical image generated by the optical active structure in each of the wavelength bands blue, green and red, that each detector comprises at least one sensor element, such that at least one sensor element uniquely defines a pixel of the optical image, the area of the pixel being substantially determined by the area of the separate defining sensor element, that each detector in one of the image planes is adapted for defining a sample of the optical image in the wavelength band corresponding to this image plane and with a number of pixels in each sample determined by the number of sensor elements in the defining detector, the digital image optimally being generated as an RGB colour image with a spatial resolution determined by the number of pixels in distinct, by the sensor elements defined positions in the optical image.
Preferably according to the invention the optical active structures in this case are refractive structures with a determined chromatic aberration or diffractive structures with a determined dispersion or combinations of such structures, and particularly it is then preferred that the refractive or diffractive structures are realized as lenslets with a diameter of at most 3 mm.
Further it is in that connection according to the invention preferred that the total number of distinctly defined pixels in the optical image in one of the wavelength bands is equal to the total number of sensor elements in the detectors for this wavelength band provided in the detector device such that in this case a one-to-one relation between a given pixel and its defining sensor element is present, whereby the digital RGB colour image can be generated with a full sampling of the optical image in each wavelength band and with three times oversampling of the whole optical image in colours.
Finally, a method for digital electronic formatting of a recorded full-format image according to the invention characterized by generating a section or field of the full-format digital image by substantially continuous or stepwise radial or axial contraction of the image towards respectively a convergence point or a convergence axis in the image, the contraction of the image taking place digitally in a data processor provided in the processor device and according to one or more determined pixel-subtracting protocols and being effected by an in-camera or externally provided operating device which is manoeuvred manually by a camera operator and automatically according to predetermined criteria, and by once again expanding formatted field radially or axially in this way stepwise or continuously from respectively the convergence point or from the convergence axis towards a full-format image.
Preferably, according to the invention the formatting may be visualised on the display device, the section or field caused by the formatting at any instant being displayed as a synthetic full-format image on the display device, but with a real spatial resolution given by the corresponding pixel subtraction value of the formatting.
Further, according to the invention a digital electronic zoom function may advantageously be implemented in the camera by the radial contraction or expansion, the field format being determined as respectively a telephoto, wide angle or macro format depending on the distance between the scene and the image plane in the camera, and by a digital electronic pan function being implemented in the camera by the axial contraction or expansion.
Further features and advantages of the present invention are disclosed by the remaining appended dependent claims.